Industrial robot system including action planning circuitry for temporary halts

ABSTRACT

An industrial robot system may enable a reduction in an installation/adjustment period, and an increase in a no-error continuous operation period. The system includes an action planning section for temporary halts, an error-inducing-task restraining section, a section for teaching task, an operation mastering section, a hand library, an optimum-task-operation generating section, a specific task library, an error-recovery-task teaching section, an error recovery library, a finger-eye-camera measurement section including an omnidirectional mirror, a three-dimensional recognition section, a controller, a manipulator, and a hand with a plurality of fingers.

TECHNICAL FIELD

The present invention relates to an industrial robot system used for aproduction system for carrying out a product assembly of an object to bemanufactured.

BACKGROUND ART

As a major issue to be addressed when a robot is introduced into aproduction line, a prevention of a temporary halt (a state in which afacility stops or idles due to a temporary trouble) has conventionallybeen pointed out. The temporary halt constitutes an obstacle to areduction of a teaching period during the startup/adjustment of aproduction line, and also constitutes an obstacle to an unattendedcontinuous operation.

As general design steps for a product, a structural design of theproduct to be manufactured and a layout design of cells used formanufacturing the product by an unattended operation are first carriedout. As a result, part connection information (part configuration treediagram) representing an order relation of connecting parts constitutingthe product, product design data such as geometric shape data of theparts, facility layout data within cells, and production facility datasuch as specifications of robots are obtained. Then, programming,installation/adjustment, and teaching tasks for operating each of thefacilities such as robots in the production system start.

As a conventional industrial robot system, a technology involvingdetecting an error occurrence and generating a recovery sequence inorder to recover a production facility is proposed for a case in whichan operation state can be represented as a binary state (such as ON/OFF)(see Patent Literatures 1 and 2, for example).

Similarly, in order to recover from a failure in a production line,there is proposed a technology in which a control sequence program isconstructed in advance block by block, and detection of a failure andactivation of a program for recovery are carried out for each block (seePatent Literature 3, for example).

CITATION LIST Patent Literature

-   Patent Literature 1: JP 3195000 B2-   Patent Literature 2: JP 6-190692 A-   Patent Literature 3: JP 3-116304 A

SUMMARY OF INVENTION Technical Problem

In the conventional industrial robot system, a processing operation forcalculating an error recovery method is carried out in advance only forthe binary state of each of components of a production system fromconfiguration information on the production system, and there has thusbeen a problem that a failure which occurs in the production systemincluding robots which can take complex multi-value (ternary or higher)state values cannot be solved.

The present invention has been made in order to solve theabove-mentioned problem, and it is therefore an object of the presentinvention to provide an industrial robot system which reduces a teachingperiod (installation/adjustment period) upon startup/adjustment of aproduction system (production line) using industrial robots, and extendsa continuous operation period in a no-error state after the start of theoperation.

Solution to Problem

According to the present invention, there is provided an industrialrobot system including a robot having a manipulator and a hand, and usedfor a production system for assembling a product which is an object tobe manufactured, including: an action planning section for temporaryhalts for generating task information and a first work path in order toaddress a temporary halt which constitutes an obstacle to a teachingtask when a production line is started up and adjusted, and anunattended continuous operation; an error-inducing-task restrainingsection for generating error information used for restraining a taskinducing an error based on the task information; a section for teachingtask for generating a second work path based on the first work path andthe error information; an operation mastering section for generating athird work path optimized for the robot based on the second work path; ahand library formed by associating an assembly task of the robot andcontrol software with each other; an optimum-task-operation generatingsection for generating a operation sequence of specific tasks; aspecific task library for storing the operation sequence of specifictasks; an error-recovery-task teaching section for teaching an errorrecovery task according to an error state based on an operation historyin the section for teaching task; an error recovery library for storingthe error recovery task; a finger-eye-camera measurement section and athree-dimensional recognition section for generating operationmonitoring information on the robot, and inputting the operationmonitoring information to the error-inducing-task restraining section,the section for teaching task, and the operation mastering section; anda controller for controlling the robot based on the second work path andthe third work path, and on the operation monitoring information, inwhich: the action planning section for temporary halts generates thefirst work path based on configuration information on the productionsystem and the object to be manufactured, information stored in the handlibrary, the specific task library, and the error recovery library, andthe error information from the error-inducing-task restraining section;the error-recovery-task teaching section calculates error recoveryinformation on components including the robot based on a cause foroccurrence of the error and the operation history from the section forteaching task; and the action planning section for temporary halts, thesection for teaching task, and the operation mastering section generateprogram information including the third work path required for teachingthe robot from the configuration information on the production systemand the object to be manufactured.

Advantageous Effects of Invention

According to the present invention, it is possible to realize thereduction of the installation/adjustment period of a production lineincluding industrial robots, and the extension of the no-errorcontinuous operation period after the operation starts.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A block configuration diagram illustrating an industrial robotsystem according to Example 1 of the present invention (example 1).

FIG. 2 An explanatory diagram illustrating an example of animplementation image of an ecological interface used for Example 1 ofthe present invention (Example 1).

FIG. 3 An explanatory diagram illustrating input/output information fora section for teaching task according to Example 1 of the presentinvention (Example 1).

FIG. 4 Explanatory diagrams illustrating an overview of an optimum tacttime search in an operation mastering section according to Example 1 ofthe present invention (Example 1).

FIG. 5 An explanatory diagram illustrating input/output information forthe operation mastering section according to Example 1 of the presentinvention (Example 1).

FIG. 6 A flowchart illustrating a task analysis while consideringsupplementary tasks such as a temporary halt recovery according toExample 1 of the present invention (Example 1).

FIG. 7 An explanatory diagram illustrating input/output information foran action planning section for temporary halts according to Example 1 ofthe present invention (Example 1).

FIG. 8 An explanatory diagram illustrating a framework of an erroroccurrence risk analysis method according to Example 1 of the presentinvention (Example 1).

FIG. 9 An explanatory diagram illustrating input/output information foran error-inducing-task restraining section according to Example 1 of thepresent invention (Example 1).

FIG. 10 An explanatory diagram illustrating input/output information foran optimum-task-operation generating section according to Example 1 ofthe present invention (Example 1).

FIG. 11 A block diagram illustrating a configuration example of asupervised learning system based on deductive learning of knowledgeaccording to Example 1 of the present invention (Example 1).

FIG. 12 An explanatory diagram illustrating input/output information foran error-recovery-task teaching section according to Example 1 of thepresent invention (Example 1).

FIG. 13 An explanatory diagram conceptually illustrating a configurationof a finger-eye-camera measurement section used for Example 1 of thepresent invention (Example 1).

FIG. 14 An explanatory diagram illustrating input/output information fora sensor feedback function based on the finger-eye-camera measurementsection according to Example 1 of the present invention (Example 1).

FIG. 15 An explanatory diagram illustrating input/output information fora gripping position/attitude measurement function based on thefinger-eye-camera measurement section according to Example 1 of thepresent invention (Example 1).

FIG. 16 An explanatory diagram illustrating, along with a block diagram,an example of three-dimensional measurement/recognition processingaccording to Example 1 of the present invention (Example 1).

FIG. 17 An explanatory diagram illustrating input/output information fora three-dimensional recognition section according to Example 1 of thepresent invention (Example 1).

FIG. 18 An explanatory diagram illustrating input/output information foran algorithm design support section according to Example 1 of thepresent invention (Example 1).

FIG. 19 An explanatory diagram graphically illustrating a concept of ahand library according to Example 1 of the present invention (Example1).

FIG. 20 An explanatory diagram conceptually illustrating an industrialrobot system according to Example 2 of the present invention (Example2).

FIG. 21 An explanatory diagram illustrating input/output information fora hand library function of an offline teaching section according toExample 3 of the present invention (Example 3).

FIG. 22 An explanatory diagram illustrating input/output information foran error detection function of the offline teaching section according toExample 3 of the present invention (Example 3).

FIG. 23 An explanatory diagram illustrating input/output information foran error-recovery-level determination function of the offline teachingsection according to Example 3 of the present invention (Example 3).

FIG. 24 An explanatory diagram illustrating input/output information foran error-occurrence-risk analysis function of the offline teachingsection according to Example 3 of the present invention (Example 3).

FIG. 25 An explanatory diagram illustrating input/output information foran action control function of an error recovery section according toExample 5 of the present invention (Example 5).

FIG. 26 An explanatory diagram illustrating input/output information fora function of enhancing a section for teaching task of the errorrecovery section according to Example 5 of the present invention(Example 5).

FIG. 27 An explanatory diagram illustrating input/output information fora teleoperation function of the error recovery section according toExample 5 of the present invention (Example 5).

FIG. 28 An explanatory diagram illustrating input/output information foran object recognition function for part picking of a recognition sectionaccording to Example 6 of the present invention (Example 6).

FIG. 29 An explanatory diagram illustrating input/output information fora hybrid vision correction function of the recognition section accordingto Example 6 of the present invention (Example 6).

FIG. 30 An explanatory diagram illustrating input/output information fora vision function for error detection of the recognition sectionaccording to Example 6 of the present invention (Example 6).

FIG. 31 An explanatory diagram illustrating input/output information fora recognition application building support function of the recognitionsection according to Example 6 of the present invention (Example 6).

FIG. 32 A block configuration diagram of an industrial robot systemaccording to Example 7 of the present invention (Example 7).

DESCRIPTION OF EMBODIMENTS Example 1

FIG. 1 is a block configuration diagram illustrating an industrial robotsystem according to Example 1 of the present invention.

The industrial robot system in FIG. 1 includes product design data andproduction facility data 1 (including part connection information,geometric shape data, and facility layout data) produced and prepared inadvance by three-dimensional CAD, a specific task specification 2produced and prepared in advance, and a robot system 3 installed in aproduction line.

Moreover, the industrial robot system includes, as components relatingto the production design data and production facility data 1, thespecific task specification 2, and the robot system 3, an actionplanning section 4 for temporary halts, an error-inducing-taskrestraining section 5, a section 6 for teaching task, an operationmastering section 7, a hand library 8, a specific task library 9, anerror recovery library 10, an optimum-task-operation generating section11, an error-recovery-task teaching section 12, a controller 30, amanipulator 31, a finger-eye-camera measurement section 32, athree-dimensional recognition section 33, and a manipulation devicegroup 34.

The controller 30, the manipulator 31, the finger-eye-camera measurementsection 32, and the three-dimensional recognition section 33 areprovided in the robot system 3. The finger-eye-camera measurementsection 32 and the three-dimensional recognition section 33 constitutean image measurement/recognition section.

Moreover, the manipulation device group 34 is provided in themanipulator 31, and the manipulation device group 34 includes universalhands and the like.

The hand library 8 stores operation sequences of gripping tasks inputfrom the manipulation device group 34.

The specific task library 9 stores operation sequences of specific tasksinput from the optimum-task-operation generating section 11 based on thespecific task specification 2.

The error recovery library 10 stores recovery task paths according toerror states generated by the error-recovery-task teaching section 12based on causes for error occurrence and operation histories ofoperators provided from the section 6 for teaching task.

The action planning section 4 for temporary halts produces a worksequence diagram including an error recovery sequence and a work pathincluding approximate coordinates based on the part connectioninformation, the geometric shape data, and the facility layout data outof the product design data and production facility data 1, and thestored information of the respective libraries 8 to 10, and inputs thework sequence diagram including the error recovery sequence and the workpath including the approximate coordinates to the section 6 for teachingtask.

Moreover, the action planning section 4 for temporary halts is mutuallyassociated with the error-inducing-task restraining section 5 to inputcandidates of a task order to the error-inducing-task restrainingsection 5, and acquire error occurrence probability information from theerror-inducing-task restraining section 5.

The error-inducing-task restraining section 5 is mutually associatedwith the section 6 for teaching task to input the error occurrenceprobability information and the cause for error occurrence to thesection 6 for teaching task, and acquire a teaching path from thesection 6 for teaching task.

Moreover, the error-inducing-task restraining section 5 generates theerror occurrence probability information and the cause for erroroccurrence based on operation monitoring information from the controller30 of the robot system 3.

The section 6 for teaching task generates the cause for error occurrenceand the operation history of the operator based on the work sequencediagram (including the error recovery sequence) and the work path(including approximate coordinates) from the action planning section 4for temporary halts, the error occurrence probability information andthe cause for error occurrence from the error-inducing-task restrainingsection 5, and the operation monitoring information from the robotsystem 3 (finger-eye-camera measurement section 32 and thethree-dimensional recognition section 33), and also generates a refinedwork path (robot program before mastering) for input to the operationmastering section 7 and the controller 30.

The operation mastering section 7 generates an optimized work path(robot program after the mastering) based on the work path from thesection 6 for teaching task and the operation monitoring informationfrom the robot system 3, and inputs the optimized work path to thecontroller 30.

The controller 30 provides control for driving the manipulator 31 andthe manipulation device group 34 based on the respective robot programsbefore and after the mastering, and the operation monitoring informationfrom the finger-eye-camera measurement section 32 and thethree-dimensional recognition section 33.

The finger-eye-camera measurement section 32 and the three-dimensionalrecognition section 33 monitor operations of the manipulator 31 and themanipulation device group 34, and input the operation monitoringinformation to the controller 30 and the section 6 for teaching task.

A description is now given of an operation according to Example 1 of thepresent invention illustrated in FIG. 1.

As general design steps for a product by a designer, the structuraldesign of a product to be manufactured, and the layout design for cellsused to manufacture the product are first carried out as describedabove.

As a result, the part connection information (part configuration treediagram) representing an order relation of connecting parts constitutingthe product, the product design data such as the geometric shape data ofthe parts, the facility layout data within the cells, and the productionfacility data such as specification of the robots are obtained.

The system operation according to Example 1 of the present inventionstarts from a state in which these results of design tasks by a designerare available.

A description is now given of the system operation according to Example1 of the present invention when the production facility starts up.

In a first phase, the product design data and production facility data 1are input to the action planning section 4 for temporary halts.

As a result, a product production task is decomposed into a sequence ofsmaller tasks, and each of these tasks are assigned to each of thefacilities in the cell, and a task order is generated based on the partconnection information in the action planning section 4 for temporaryhalts. On this occasion, in decomposing the task and determining thetask order, if a candidate of the task order is given to theerror-inducing-task restraining section 5, the error occurrenceprobability information for the given task is returned, and a task orderhaving a low risk of temporary halt is thus selected. It should be notedthat the error occurrence probability is updated at any time by theoperation monitoring information while production is being carried out.

It should be noted that the temporary halt refers to a state in whichwhile an automatic/semiautomatic task such as transporting, machining,assembly, or inspection/measurement is being applied to an intendedobject of task (such as a raw material, a workpiece, and a machiningtool) in an automated facility, an abnormal state arises on the intendedobject of task or a facility portion, and task functions of the facilitytemporarily halt while the temporary halt is not serious.

Moreover, the respective tasks and the task order are determined, and a“work sequence diagram including an error recovery sequence” including acheck point for examining a temporary halt during the task, a recoverypoint at which the task can be resumed when the check point is notpassed, a recovery path for returning to the recovery point, a via pointfor avoiding an obstacle, work paths connecting the respective points toeach other, a sequence describing the execution order of the respectivepaths, and a synchronization point at which other robots and devices arecaused to wait is generated in the action planning section 4 fortemporary halts.

Moreover, an attribute label is attached to each of the work paths.

The labels may include “movement between two points”, “movement to viapint”, “task accompanying action by end effector device”, “taskaccording to sensor feedback control such as approaching movementimmediately before/after gripping part or task operation optimized inadvance”, and “during recovery sequence from temporary halt”. It shouldbe noted that a plurality of labels may be attached to a single workpath.

On this stage, the work sequence diagram includes only the labeled workpaths, and the respective points, and does not include contents of eachof the work paths.

It should be noted that the content of the work path includes positionand attitude coordinates at both ends of the path and via points (aplurality of via points are added according to necessity) and aspecification of method of movement between the coordinates (such as acontrol law and an interpolation method).

Moreover, the task operation optimized in advance includes tips forrestraining task errors, and quickly and flexibly carrying out a task.

Further, task operations including a gripping strategy obtained from astructure of a hand optimized based on an attitude for gripping a part(hand library 8), specific tasks such as the peg-in-hole (use ofhuman-type flexible fingers) frequently employed in a production process(optimum-task-operation generating section 11 and specific task library9), and recovery paths for returning to a recovery point according to anerror state (error-recovery-task teaching section 12 and error recoverylibrary 10), namely task operations which have been obtained asknowledge from results of teaching and execution of past robot tasks areused for generating work path candidates in the action planning section4 for temporary halts.

As a result, it is possible to restrain a past task inducing a temporaryhalt from occurring again.

As a next phase, rough contents of the work paths for each of the tasksare generated by using the facility layout data within the cells and thetask order data in the action planning section 4 for temporary halts.

For example, if there are a part placement location and a work bench ina production cell, and a work path of a specific trajectory fortransporting a part is considered for a task for transporting the partfrom the part placement location to the work bench, a robot mayinterfere with surrounding objects, and it is eventually necessary toset precise values to instances of each of the work paths.

Each of the work paths is generated at a precision of approximately 5 cmon this occasion, and a reference attitude for gripping an object(relative attitude between the part and the hand) is determined in thesection 6 for teaching task on a later stage by an operator usingteaching task input means.

The above-mentioned operation is repeated for all the work paths, anddata including “work sequence diagram including error recoverysequences+work paths including approximate coordinates” is obtained asthe output information from the action planning section 4 for temporaryhalts up to this phase.

The section 6 for teaching task then starts an operation.

In the section 6 for teaching task, refined work paths to which absolutecoordinates are specified are determined by a teaching operator usingteaching task input means only by carrying out final positioning ofimportant operation points such as gripping points for the work pathsincluding approximate coordinates.

A user interface based on the ecological interface theory displayed on apersonal computer for the teaching task or a teaching pendant presentsimportant operation points and task states on this occasion, and theoperator carries out the refining task for the position/attitude, andadds work paths according to necessity on the teaching task input meanswhile observing the presented states.

The task state is acquired by a camera having an omnidirectional fieldof view provided at a fingertip (manipulation device group 34) of therobot (manipulator 31) in the finger-eye-camera measurement section 32.

An image acquired in this way spans a wide range hardly causing anocclusion in the field of view, and hence the operator easily observesthe state of the fingertips of the robot during the manual operation.

Alternatively, the task state is acquired by a force sensor (not shown)provided at the fingertip (manipulation device group 34) of the robot(manipulator 31).

The force sensor is a sensor for measuring both a force and a torqueapplied to the sensor, and the fingertip of the robot is a finger partof the robot hand or a connection portion between the robot hand and therobot. The same effect is also provided even if a torque sensor isprovided for each of axes of joints of the manipulator, and both theforces and torques applied to an arbitrary position of the robotfingertip are estimated by using data acquired from the plurality oftorque sensors.

The data acquired by the force sensor can discriminate a smalldisplacement of the robot fingertip position which cannot bediscriminated by the camera if the robot fingertip comes in contact withthe intended object of task, and the operator thus more easily observesthe state of the fingertip of the robot during the manual operation. Inother words, the robot fingertip position can be finely adjusted.

In other words, information obtained by matching the data acquired bythe force sensor against a model obtained by abstracting a phenomenonoccurring on the robot fingertip and prepared in advance, or byassigning the data acquired by the sensor to a model represented byequations to perform calculation is displayed as a phenomenon occurringon the fingertip on an image.

As a result, the operator easily observes the state of the fingertips ofthe robot during the manual operation.

Moreover, a semi-autonomous mode in which, while a certain attitude isautonomously maintained with respect to an intended object, aninstruction to move in a certain direction is given to other axes, or aninstruction by an operator is reflected on other axes is realized bycausing the finger-eye-camera measurement section 32 to process theacquired image information, thereby acquiring information on normallines of gripped surfaces of the intended object, which contributes to areduction in load of the teaching task.

Similarly, if the state of the fingertip of the robot is observed by theforce sensor, a semi-autonomous mode in which, while a certain attitudeis autonomously maintained with respect to an intended object, aninstruction to move in a certain direction is given to other axes, or aninstruction by the operator is reflected on other axes is realized byacquiring information on normal lines of contact surfaces of theintended object and the robot fingertip, which contributes to areduction in load of the teaching task.

The error-inducing-task restraining section 5 is also used in this case,and if an operation inducing a task mistake is taught, a warning isgenerated.

Moreover, it is necessary to determine, for example, the grippingposition/attitude in the section 6 for teaching task, and thus theoperator does not simultaneously operate “six degrees of freedom”.Instead, the system takes over the complex simultaneous control of themultiple degrees of freedom, and the operator operates only a smallnumber of degrees of freedom.

As a result, there is provided a function of determining the grippingposition/attitude while reducing the load.

The refined work paths are then optimized, and work paths (aftermastering) are work paths which present smooth work operations, andshort tact times are acquired in the operation mastering section 7.

As described above, the action planning section 4 for temporary halts,the error-inducing-task restraining section 5, the section 6 forteaching task, and the operation mastering section 7 generate robotprograms for the robot system 3 in relation to the respective pieces ofdesign data 1, the specific task specifications 2, the respectivelibraries 8 to 10, the optimum-task-operation generating section 11, andthe error-recovery-task teaching section 12.

In other words, robot programs executable on the controller 30 includingrecovery sequences generated if a temporary halt occurs can be obtainedfrom the product design/production facility data 1 while the load on theteaching operator is reduced largely compared with a conventional case.

A description is now given of a system operation performed when theproduction according to Example 1 of the present invention is carriedout.

A robot operation sequence performed when a product is produced isrepresented by the following items (1) to (6).

(1) Normally, work paths described in robot programs are sequentiallyfollowed.

On this occasion, the distance to and the attitude with respect to anintended object are measured in real time using the finger-eye-camerameasurement section 32 and the three-dimensional recognition section 33,and the task is carried out according to sensor feedback control inorder to absorb a variation of tolerance in part dimension and avariation in positioning, thereby realizing a stable operation.

Moreover, a high-speed, high-precision robot operation is realized bymeans of, for example, a feedback of image/distance hybrid type using atwo-dimensional image feedback having a high processing rate and athree-dimensional distance/attitude data feedback having a slightlylower processing rate.

Moreover, the finger-eye-camera measurement section 32 can measure thegripping position and attitude of a part while the part is gripped, anda correction of errors in position and attitude which are generated whenthe part is gripped, and a precise positioning by re-gripping can thusbe enabled, thereby restraining a temporary halt caused by adisplacement.

(2) A plurality of robots or devices are synchronized at asynchronization point.

(3) An examination is carried out at a check point, and the sequencereturns to (1) if no abnormality is found.

(4) If the examination is not successful at the check point, a recoveryoperation is carried out.

(5) The robot returns from an arbitrary position between check points toa recovery point following a recovery path given in advance in therecovery operation, and returns from that point to (1), thereby resumingthe operation in a normal mode.

(6) If the robot cannot reach the recovery point after repeating therecovery operation a predetermined number of times or more, the operatoris called.

A teaching method for the recovery operation by the operator is recordedin the error-recovery-task teaching section 12, and is utilized asknowledge for producing the robot program next time.

A detailed description is now sequentially given of the section 6 forteaching task, the operation mastering section 7, the action planningsection 4 for temporary halts, the error-inducing-task restrainingsection 5, the optimum-task-operation generating section 11, theerror-recovery-task teaching section 12, the finger-eye-camerameasurement section 32, the controller 30 (image processing section),the three-dimensional recognition section 33, the manipulation devicegroup 34 (universal hand), and the hand library 8 in relation to thebackground art.

First, a detailed description is given of the section 6 for teachingtask of FIG. 1.

A portion of a teaching task for a robot in which automation isdifficult to be attained such as a portion requiring decision making byadjustment on site is responsible for a human operator.

The task is thus supported by providing an effective user interface inthe section 6 for teaching task.

As described above, though the “work sequence diagram including errorrecovery sequences” and the “work paths including approximatecoordinates” are acquired as intermediate outputs of the system throughthe processing by the action planning section 4 for temporary halts, itis necessary to add/modify a specific shape and precise position data ofvia points of the work path according to actual facilities and parts tobe assembled (workpieces) in order to complete the “work sequencediagram including error recovery sequences” and the “work pathsincluding approximate coordinates” as robot programs.

It is thus necessary to consider various requirements such as areduction in the tact time, avoidance of interferences with otherobjects, and countermeasures for task phases high in probability ofinducing errors in the refinement task of the robot work path.

Particularly, the most important problem lies in teaching in anoperation phase in which a physical contact occurs between differentobjects including a hand such as gripping and assembling a part in orderto realize a robust work path for a robot against various variationfactors.

Though it is necessary to precisely lead the robot to properposition/attitude in order to teach correct gripping and assembling of apart, determination therefor solely depends on visual recognition of theoperator, and is thus a point which reflects skills of the operator.

Moreover, there are variations within tolerance in the dimensions andshapes of a part, and it is thus necessary that a sample and a fixedstate thereof used in the teaching task properly represent variations ofthe part during a continuous operation. Otherwise, uncertainty which isnot expected in the taught work path largely increases an erroroccurrence probability.

The various sensors (finger-eye-camera measurement section 32,three-dimensional recognition section 33, and force sensor (not shown))constituting a robust recognition intelligent section are utilized as aninformation source for teaching a robot work path, and measured data(operation monitoring information) acquired from the sensors arevisualized in a proper form in the section 6 for teaching task ascountermeasures against the problems relating to the teaching task assummarized before.

As a result, it is possible to change the determination made for preciseposition and the attitude to a more simple and easy determination,thereby promoting efficiency of the teaching task.

Moreover, a countermeasure in the teaching task phase against thevariation factors in the assembly task is realized by similarlyvisualizing information on the variation in the dimension and the shapeof the parts to be assembled in the section 6 for teaching task.

If many pieces of numerical data are to be presented on atwo-dimensional screen, it is necessary to integrate these pieces ofdata by a form and an arrangement suitable for the teaching task so thatthe operator intuitively understands a proper task state. The theory ofthe “Ecological Interface Design (EID)” may be used as a basis for thispurpose.

FIG. 2 is an explanatory diagram illustrating an example of animplemented image of the ecological interface.

In FIG. 2, a mutual relationship between a universal hand 34 a of themanipulator 31 and a part W to be assembled is acquired as various typesof sensor data, and is visualized and presented as information on thepart W to be assembled.

Employment of the operation sequence of specific tasks such as the pegin hole which can be optimized in advance as a template for refining arobot work path is very efficient for reducing labor required for theteaching task.

The section 6 for teaching task thus refers to a operation sequence ofspecific tasks via the specific task library 9, thereby enabling theoperator to use the specific task library 9 for refining a work path.

Moreover, the section 6 for teaching task communicates with theerror-inducing-task restraining section 5, always checks whetherteaching of a task operation which possibly induces an error is present,warns the operator if corresponding teaching is carried out, andsupports execution of an efficient teaching task.

Action generation is carried out always based on a relative relationshipbetween an intended object of task and a surrounding environment in theteaching task for a robot. It is conceived that there are many cases inwhich a semi-autonomous mode in which sensor feedback is carried out forcertain degrees of freedom while an operation by an operator is followedfor other degrees of freedom is efficient on this occasion.

Though the information from the image sensor, the distance sensor, andthe force sensor are exemplified as expected representative sensorinformation, if other sensor information is necessary depending onintended purpose of the task, a system function corresponding theretomay be added.

Input/output information for the section 6 for teaching task isillustrated in an explanatory diagram of FIG. 3.

Specifically, the input information includes the work sequence diagramincluding error recovery sequences, the work path including approximatecoordinates, the various types of sensor data, and information on a partto be assembled.

Moreover, the output information includes the refined work pathscontaining a start point coordinate, an endpoint coordinate, and rangesand resolutions of a plurality of via point coordinates of the workpaths.

A detailed description is now given of the operation mastering section 7of FIG. 1.

A task operation plan for a vertically articulated robot presentscomplexity caused by high degrees of freedom of a mechanism. Forexample, even if a trajectory connecting between only a start point andan end point is derived, it is necessary to solve the two-point boundaryvalue problem, and it is difficult to solve this problem eitheranalytically or numerically. Moreover, it is more difficult to optimizea trajectory including a plurality of via points in addition to thetrajectory connecting between the start point and the end point.

A method of searching for an optimum trajectory including via points fora vertically articulated robot is thus used in Example 1 of the presentinvention. This is a method of searching for a semi-optimum solutionfrom observed signals of an operation response obtained by operating anactual robot.

As a result, it is expected that versatility is provided for flexiblyadapting to a change of and an increase in complexity of a taskenvironment. As a result, a tact time can be reduced when the sameoperation is repeated.

FIG. 4 are explanatory diagrams illustrating an overview of an optimumtact time search in the operation mastering section 7.

FIG. 4(a) illustrates an example of a work path of the robot(manipulator 31) routing from a start point (−60, 30, 80, 0, 0, 0) to anend point (60, 30, 80, 0, 0, 0) including one via point (0, J2, J3, 0,0, 0).

Moreover, FIG. 4(b) illustrates the position of an obstacle D withrespect to the robot (manipulator 31) in three views (plan view, sideview, and front view).

The via point (0, J2, J3, 0, 0, 0) of FIG. 4(a) is set so as to avoidthe obstacle D of FIG. 4(b).

Further, FIG. 4(c) illustrates a concept of the optimum tact time searchin three-dimensional perspective view, and respective axes correspond toa joint angle “J2”, a joint angle “J3”, and the tact time. “J2” and “J3”are included in the coordinates of the via point.

If the via point of FIG. 4(a) is changed in various ways, a tact time isobserved for each via point, and a curved surface illustrated in FIG.4(c) is acquired.

Work paths interfering with the obstacle D are represented as a tacttime “0” in FIG. 4(c). On this occasion, the shortest tact times appearin a zone Z1, which is illustrated as a blank in the curved surface.Though a case for only one via point at which only the two joints moveis described for the sake of simplicity in FIG. 4(c), a plurality of viapoints at which a plurality of joints simultaneously change constitutethe work path in reality, and it is thus necessary to search amulti-degree of freedom space for the blanked area Z1. Moreover, asearch path PS1 having a small number of trials and search paths PS2 andPS2 having a large number of trials are illustrated as thick arrowlines.

A system function for increasing the speed of the search as describedabove is provided according to Example 1 of the present invention.

Specifically, a framework of the Active Learning in ComputationalLearning Theory is applied. In other words, the most preferable trial tobe tried next time is optimized by using data of results of past trialswhen a search trial is repeated one after another.

In other words, for a plurality of candidates which can be tried nexttime, an operation of converting results which are considered to beacquired by trying the respective candidates into numbers, and comparingthe numbers with each other, thereby determining an order of thecandidates to be tried starting from the next time is carried out. Forthe conversion of the result into a number, a model for estimating aresult to be obtained if a certain candidate is tried is built by usingdata obtained from past trials, and the candidate considered to beacquired by trying the certain candidate is then converted into a numberfor each of all the candidates using this model.

The optimization of the candidate using the acquired number may becarried out for each trial or each of a plurality of trials. It is knownthat an effect of reducing the number of trials is increased by carryingout the optimization for each time.

Actually, if a certain candidate was used for trial, an acquired resultis combined with results acquired up to this time. In other words, themodel built during the optimization becomes gradually precise as thetrial is repeated.

As a result, the number of the trials decreases as a whole, and the“search path PS1 having a small number of trials” is realized asindicated by the “thick arrow line” of FIG. 4(c).

Moreover, if a vibration of the robot arm is large, a next operationcannot be started until the vibration attenuates. If this waiting timebecomes long, the tact time becomes long, and it is thus conceived thatthe attenuation of the vibration is an object to be considered for thetact time reduction.

Though this vibration is caused by an inertial force of the hand/arm anda restoring force generated due to a spring effect of a speed reductionmechanism such as a harmonic drive and a transmission mechanism, aselection of a work path is an important item to be considered.

Actually, a method of discretely instructing task points and checkpoints is common in teaching in a conventional system, and a trajectoryconsidering the vibration restraint is not selected for selecting amovement path between points and a trajectory in a work path. Anunwanted vibration is thus excited each time anacceleration/deceleration occurs.

As one of conventional methods which are actually employed forrestraining vibration, a notch filter fixed to a natural frequency atthe most extended state of the arm is applied to a motor control signal.However, the natural frequency changes according to a change in weightof the workpiece and the attitude of the arm, and it is thus consideredthat the filter effect is not sufficient. Moreover, a time delay uniqueto the digital filter conflicts with the reduction in the task period.

A skilled operator of a crane operates a tip of the crane so as torestrain the vibration of a workpiece as much as possible after theworkpiece is suspended until the workpiece arrives at a destination, forexample. Specifically, this operation is realized by moving the tipposition of the crane onto a perpendicular line of the workpiece at amoment at which the workpiece reaches the maximum displaced position ofa pendulum due to deceleration/acceleration, and the velocity thusbecomes zero.

This means the vibration of the workpiece can be avoided when theworkpiece stops if the target position can be set to the maximumdisplaced position of the pendulum.

The case of the industrial robot system according to Example 1 of thepresent invention is a system in which the hand and the arm are masses,the harmonic drive acts as a spring element, and motors are integrated,and is thus different from the example of the crane. However, a methodof deriving conditions under which the hand does not vibrate at a targetposition (stop position), and changing an operation trajectory of therobot and an acceleration/deceleration operation pattern is employed.Further, the hand is attached to the tip of the robot. By combining amethod of using the hand for damping the arm by causing the hand to actas a dynamic damper or an active dynamic damper, there is provided aneffect of further reducing a period required for restraining thevibration.

In other words, a work path having a short tact time can be acquired byalternately carrying out the above-mentioned two tasks in the operationmastering section 7. Moreover, the vibration restraining control of therobot arm also contributes to prevention of a decrease in thepositioning precision, and to restraint of the temporary halt.

Input/output data for the operation mastering section 7 is illustratedin an explanatory diagram of FIG. 5.

Specifically, the input information includes the start point coordinateand the end point coordinate of a work path, and the range andresolution of the coordinates of the plurality of via points, taskenvironment information, and robot information.

Moreover, the output information includes the start point coordinate andthe end point coordinate of the work path, the coordinates of theplurality of via points, a trajectory through the start point, theendpoint, and the plurality of the via coordinate points, andacceleration/deceleration instructions.

A detailed description is now given of the action planning section 4 fortemporary halts of FIG. 1.

Decomposition of a product assembly task into detailed element tasks,assignment of these element tasks to each of facilities, and specificdetermination of a work sequence and a work path are carried out in adesign of a cell production robot system.

A result of this design influences on difficulty of the teaching therobot the element tasks and error recovery processes as well as anefficiency of each of the element tasks carried out by the robot.However, it is necessary to examine items ranging from a microscopicpoint of view of increasing an efficiency of each of the element taskscarried out by the robot to a logistic point of view of the overallproduction system in order to design the cell production robot system.

Presently, it is very difficult to design a comprehensively optimumsystem. Particularly, though supplementary tasks such as processingcarried out if a robot hand is changed or a temporary halt occurs have alarge influence on the efficiency of the overall cell production robotsystem, presently there is no tool for examining the influence exertedby those supplementary tasks on an initial design state.

Moreover, though many commercial software programs for simulating arobot operation in detail have been developed, a large amount of laboris required for building a simulation model therefor, and these softwareprograms are not thus suitable for the initial design stage on which thevarious items need to be simultaneously examined.

The action planning section 4 for temporary halts is then used forquickly carrying out comprehensive evaluation of the cell productionrobot system including supplementary processes such as the hand changeand the error handling on the initial design stage according to Example1 of the present invention.

The action planning section 4 for temporary halts determines an overviewof the robot task in cooperation with each of functional configurationsections of the error-inducing-task restraining section 5 and theoptimum-task-operation generating section 11, which are described later.

A specific functional configuration of the action planning section 4 fortemporary halts is now described.

First, the action planning section 4 for temporary halts acquiresthree-dimensional geometric information on the part connectioninformation, the geometric shape data, and the facility layout data fromthe results of the product structure design and the production facilitydesign (product design data and production facility data 1). Moreover,the action planning section 4 for temporary halts identifies surfaces ofgeometrical parts which can be gripped, and decomposes a sequence ofassembly task into element tasks from those pieces of information.Further, the action planning section 4 for temporary halts generatescandidates of the work sequence, and determines facilities to which eachof the element tasks is to be assigned based on the processing resultthereof.

On this occasion, the action planning section 4 for temporary haltsinputs the candidate proposals of the work sequence to theerror-inducing-task restraining section 5 and the optimum-task-operationgenerating section 11, and receives the error occurrence probabilityinformation on the candidates of the work sequence from theerror-inducing-task restraining section 5.

Then, as illustrated in FIG. 6, the overall assembly task including thesupplementary tasks such as the change of the robot hand and therecovery operation upon the occurrence of the temporary halt is analyzedon the planning stage of the work sequence using the error occurrenceprobability information, and a robust work sequence is determinedagainst the occurrence of the temporary halt.

FIG. 6 is a flowchart illustrating the task analysis considering thesupplementary tasks such as the temporary halt recovery.

In FIG. 6, tasks t₁ and t₂ on the upstream side are carried outaccording to recovery tasks c₃ and c₄, respectively, based ondetermination of an error such as occurrence of the temporary halt.

The action planning section 4 for temporary halts proceeds to a check c₁after a task t₃ following the task t₂, and checks presence/absence ofoccurrence of a temporary halt.

If it is determined in the check that a temporary halt is present, errordecision is made, and the action planning section 4 for temporary haltsproceeds to the recovery tasks c₃ and c₄. On the other hand, if it isdetermined that a temporary halt is not present, and the task is thusnormal, the action planning section 4 for temporary halts proceeds to atask t₄, and a task t₅ is carried out after a hand change c₂.

As a result, the work sequence including supplementary tasks such as thetiming for the hand change and the return from the temporary halt isorganized according to the statistical evaluation based on the erroroccurrence probability information

It is desired that the cooperation with the error-inducing-taskrestraining section 5 be carried out in real time during the designing.

Moreover, a template of a work path may be acquired as input informationfrom the optimum-task-operation generating section 11 according to thetype of an element task during the organization of the work sequence ofFIG. 6. Task periods can be more precisely estimated during theorganization of the work sequence by using this information as a libraryof task operations which are formed into packages, and provideproficient skills.

An optimum approximate work path for the robot is automaticallygenerated at a precision of approximately 5 cm based on the worksequence determined as described above. The generated approximate workpaths are to be output to the detailed design stage such as the teachingfor the robot.

On this occasion, recovery points to the tasks t₁ and t₂ upon erroroccurrence, synchronization points with other facilities, and the point(c₁) for checking success/failure of the defined task are simultaneouslydetermined and output.

As a result, productivity of an overall design/implementation process ofthe cell production robot system can be largely increased byfront-loading consideration (such as consideration for a method ofaddressing a temporary halt), which has to be sequentially carried outin a post process of the design in a conventional cell production robotsystem, thereby carrying out proper decision making on an early designstage.

Moreover, the following point can be additionally described ascontribution to the teaching task.

In other words, there is provided a merit that the operator can carryout without hesitation a task which causes a robot to actually carry outonly a partial sequence, which the operator particularly wants to check,out of the overall sequence, thereby checking the robot operation.

This is because cleanup (returning physical positions of the robot andthe peripheral devices to initial positions, and resetting variables ofcontrol devices to certain initial values), which the operatorconventionally has to take time for and carefully (without makingmistakes) carry out, is automated by employing the error recoverysequence.

Input/output information for the action planning section 4 for temporaryhalts is illustrated in an explanatory diagram of FIG. 7.

Specifically, the input information includes the three-dimensionalgeometric information on the part connection information and thegeometric shape data of the product, and the facility layout data, theerror occurrence probability information for the candidates of the worksequence, and the templates of the work path.

Moreover, the output information includes the work sequences includingthe supplementary tasks, the approximate work paths, points requiringthe teaching task, recovery procedures upon error occurrence, therecovery points, the synchronization points, and the part surfaces whichthe robot can grip.

A detailed description is now given of the error-inducing-taskrestraining section 5 of FIG. 1.

The error-inducing-task restraining section 5 analyzes a risk of erroroccurrence during the robot task, provides information necessary forerror reduction, and restrains robot operations inducing errors frombeing set (or being taught) during the task design or the teaching taskcarried out by the operator.

Specifically, during the task design, the error-inducing-taskrestraining section 5 estimates risks of the errors (magnitude ofinfluence of and likelihood of error occurrence) which possibly occur inthe set work sequence, and supports the action planning section 4 fortemporary halts and the optimum-task-operation generating section 11 insetting check points for checking whether or not a task sub-goal isattained for detecting error occurrence and selecting a task methodwhich rarely induces an error (such as starting assembly from the top orbottom).

Moreover, the error-inducing-task restraining section 5 investigates acause of an error example during the operation, which is detected at acheck point, thereby supporting improvement of recovery points andrecovery methods in an error recovery method.

Simultaneously, knowledge relating to the error occurrence is formedinto a database, thereby supporting future study of errorcountermeasures.

Moreover, during the teaching task by the operator, theerror-inducing-task restraining section 5 warns the section 6 forteaching task that an operation to be instructed next time to the robothas a high risk of generating an unfavorable phenomenon, and teaching isat an entry to a task group having a high risk (or is already present inthe middle thereof) as the task by the robot progresses, therebyrestraining the task setting which tends to induce an error.

First, there is an “error occurrence analysis method” of identifying anoccurrence process of a “system error” which should be avoided by thecell production system using the robots in order to realize theabove-mentioned function. This is an analysis in the opposite directionfor determining a cause of the occurrence from a system error which isan effect.

On the other hand, the error occurrence risk estimation is an analysisin a forward direction for estimating system error occurrence from taskconditions, and estimates how the error occurrence risk successivelychanges as the task progresses.

It is possible to refer to system engineering methods used for thesystem risk analysis for consistently carrying out the analysis in bothdirections.

While a dependency between task characteristics and the error occurrenceprobability are being considered using a Bayesian network, the risks oferror occurrence are successively updated based on task informationadded as the task progresses.

Further, the degree of details of the design information to be used isdifferent between the early stage and the teaching stage for thedesigning of a task to be carried out by the robot, and thus ahierarchical analysis method according to the stage is used to enable aconsistent analysis starting from qualitative information toquantitative information.

Moreover, basic error-relating information relating to system componentsis accumulated as information source such as the failure mode andeffects analysis so that the basic error-relating information is usedagain by other functional configuration sections for examining the errorrecovery method and the like.

FIG. 8 is an explanatory diagram illustrating a framework of an erroroccurrence risk analysis method (risk analysis result of erroroccurrence), and illustrates a causal relationship between erroroccurrence and causes as a Bayesian network.

In FIG. 8, M stages of task 1 to task M proceed following a sequence:task 1→task 2, . . . , →task M, and respectively have influentialrelationships (refer to dotted arrow lines) with N of a cause 1 to acause N.

Moreover, each of the cause 1 to the cause N has causal relationshipswith n of an occurrence condition 1 to an occurrence condition n_(i)relating to an error i, and leads to occurrence of the error i.

If a risk of error occurrence is estimated, estimation processing iscarried out from tasks (lower stage side of the figure) to theoccurrence of errors (upper stage side of the figure).

Conversely, if a cause of error occurrence is identified, identificationprocessing for a cause is carried out from occurrence of an error (upperstage side of the figure) to tasks (lower stage side of the figure).

Input/output information for the error-inducing-task restraining section5 is illustrated in an explanatory diagram of FIG. 9.

Specifically, the input information includes a work sequence diagramillustrating work sequences, the work sequence diagram including errorrecovery sequences, information such as basic design specifications ofthe facilities and the robots, and the teaching data by the operator.

Moreover, the output information includes transition information on theerror occurrence probability based on the work sequence, estimatedresult/warning of error occurrence probability, and information oncauses/countermeasures for error occurrence.

A detailed description is now given of the optimum-task-operationgenerating section 11 of FIG. 1.

In the assembly task carried out by a robot, a “part assembly” phase inwhich a part held by the hand and a workpiece to which the part is to beassembled can physically contact with each other is one of the phases inwhich a trouble leading to a temporary halt or the like is most likelyto happen.

Moreover, even for absolutely the same part or workpiece, it is knownthat a large difference in frequency of trouble generated in the “partassembly” phase often occurs between a work path taught by a proficientengineer and a work path taught by a person who is not proficient.

This is because there are “proficient skills” for realizing the “partassembly” phase in a simple and robust way even for the assembly taskcarried out by a robot, and may be interpreted that a work pathreflecting the proficient skills hardly causes troubles.

The optimum-task-operation generating section 11 outputs templates ofoptimum work paths reflecting the “proficient skills” for a certainnumber of typical assembly process patterns (hereinafter, each ofpatterns of assembly process is referred to as “specific task”).

As a result, in the section 6 for teaching task, a robust work path canbe easily taught by referring to the template of the optimum work pathwithout depending on experiences of proficient engineers.

Example 1 of the present invention is a method of generating offline theabove-mentioned template of the optimum work path for each of thespecific tasks according to the gripping attitude of a part forrealizing the above-mentioned teaching.

There are various errors in an assembly task by a robot such as taskerror/teaching error which occurs when the manipulator 31 operates thehand, relative position/attitude errors between the hand and a part,dimension/shape errors of a part/workpiece, and position/attitude errorsof a workpiece, and it is conceived that a nature of these errors asprobability variables generates uncertainty of a task result.

A basic idea of Example 1 of the present invention is a point that theabove-mentioned “proficient skills” are considered as “skillfulnessadapting to the uncertainty of those errors”. Then, a scale (entropy)for quantitatively evaluating the uncertainty generated by an error isintroduced, and comparison among different work paths is enabled basedon how the entropy changes in the course of the work path.

An algorithm which constructs a work path template based on which therobot can skillfully handle the uncertainty according to inputinformation is acquired by developing an approach for constructing (orsearching for) a work path in which the uncertainty can be skillfullyhandled.

Further, the work path is further improved by adding a logic ofsearching a neighborhood to the work path template obtained by this“work path template construction algorithm”.

As a specific procedure, “a process of inserting a part into a hole or aslot of a workpiece” as a pattern of the assembly process which is mostoften seen on an actual manufacturing site is first taken as thespecific task.

This insertion process varies in nature according to the shape of apart, and thus the process for a so-called peg-in-hole, which is themost general shape, is considered in the first step, and a shape for theprocess gradually extends to a more complicated shape.

Input/output information for the optimum-task-operation generatingsection 11 is illustrated in an explanatory diagram of FIG. 10.

Specifically, the input information includes the type of a specific task(such as the peg-in-hole), design information (such as a dimension and ashape) of part/workpiece, major errors to be considered anddistributions thereof, gripping attitudes of the part (a plurality ofpossible candidates are generally given), robot information (such aspossible element operations and control laws), and available sensorinformation.

Moreover, the output information includes templates for a work pathincluding conditional branches (respectively output if there are aplurality of candidates of the gripping attitude), via points on thework path templates, and implication of element work sequences.

A detailed description is now given of the error-recovery-task teachingsection 12 of FIG. 1.

The operation of a robot recovering from an error state including atemporary halt is presently provided by an operator, and the operatorswitches a strategy of the recovery operations according to the errorstate. The operator thus learns each of the operations based onexperiencing an effect of a recovery.

Specifically, an aspect of an object to be controlled which has causedthe error state is divided into cases, and a control law for localrecovery for each of the cases is learned as a control law as general aspossible corresponding to a cause of the error by going back a statetransition which has reached to the error state, thereby identifying adirect/indirect cause of the error.

The learning is divided into major two phases on this occasion. Onephase is to group various error states, and the other phase is toacquire a general control law for recovery in each of the grouped errorstates. These two learning phases cannot be separated from each other,and the learning proceeds in a “co-evolutional manner” according to thelearning in each of the phases.

A “learning method from a single example” using the explanation-basedlearning (EBL), which is a representative method of the deductivelearning, is applied according to Example 1 of the present invention.

In this learning method, a system generates an explanation for why aninput of a given recovery operation sequence for a single teachingexample (namely, an error recovery taught as demonstration) can be amodel example.

Portions of the training example where teaching know-hows are utilizedare selectively extracted, and are used for generating knowledgegeneralized into an expression form (operability norm) defining astandard for applicability of the knowledge (namely, general controllaw).

On the other hand, the grouping of the error state is carried out whileassignment to the most proper typical class is successively determinedout of sequences of the error recovery operation given as the teachingexamples. A scale of category usefulness is used for the assignment onthis occasion.

An added new example (error state to be grouped) is caused totemporarily belong to a candidate typical class, an attribute valuedistribution within the class is updated, and the category usefulness isthen calculated.

A procedure of the typical class generation starts from a classincluding a single error state, and reorganization of typical classes isdetermined and carried out based on an evaluation criterion of thecategory usefulness while the following four class operations areapplied each time a new example is added to the class. Specifically, thecategory usefulness is calculated for each of the four operations:addition of an existing class; generation of a new class having a singleexample; unification of a plurality of existing classes; anddecomposition of an existing class. An operation which maximizes thecategory usefulness is then carried out. As a result, successiveoptimization for grouping and identification of an error state isenabled.

FIG. 11 is a block diagram illustrating a configuration example of asupervised learning system based on the deductive learning of knowledgedescribed above.

In FIG. 11, the supervised learning system includes a learning block forgrouping errors and a control law learning block for recoveryoperations.

The learning block for grouping errors includes N typical classes 1 to Nbranching from a repertory of errors.

Moreover, the control law learning block for recovery operationsincludes a stage of teaching recovery operations corresponding to eachof the typical classes 1 to N by the operator, a stage of inputtingsequences of the recovery operation, a stage of learning general lawsfrom one corrected example by means of the explanation-based learning,namely, EBL, N control laws 1 to N constituted of each of leaningresults of the general laws, and background knowledge (database) forgenerating explanation based on the learning results of the generallaws.

Further, the learning block for grouping errors is reconstructed inresponse to addition of an example based on the learning results of thegeneral laws.

The knowledge acquired by the supervised learning system of FIG. 11 isacquired in the expression form satisfying the operability norm, and canbe directly compared with “error states” other than the error statetaught as the demonstration.

Therefore, a recovery operation from the same grouped error state isautomatically carried out without an intervention of a teaching personby applying this control law. As a result, it is expected that thefrequency and teaching period of the conventional teaching task belargely reduced.

Input/output information for the error-recovery-task teaching section 12is illustrated in an explanatory diagram of FIG. 12.

Specifically, the input information includes a cause of each erroroccurrence, a robot operation sequence upon each error occurrence, and arecovery operation history by the operator for each error.

Moreover, the output information includes a general recovery task pathand a recovery operation sequence for a grouped error state.

A detailed description is now given of the finger-eye-camera measurementsection 32 of the image measurement/recognition section of FIG. 1.

Though there is conventionally a hand eye camera provided at a positionclose to a tip of an arm of the manipulator 31, an object of theconventional camera is to obtain an effect of moving a view point usingthe degrees of freedom of the arm rather than to observe a peripheryincluding the hand tip and an intended object, and it is assumed thatthe conventional camera observes the intended object at a certaindistance. The conventional camera does not thus have a structure forsufficiently recognizing a close/contact state between the hand tip andthe intended object or the peripheral environment, or a state of theobject being gripped, which are important in the teaching or a recoveryoperation from a temporary halt, and an occlusion caused by the armitself and the peripheral environment thus possibly occurs.

Therefore, a camera according to Example 1 of the present invention is afinger eye camera (camera for simultaneously measuring fingertip andperiphery) having an ultra-small optical system which can be disposed ata position relatively close to the fingertip of the hand of themanipulator 31, and can observe image information in a wide rangeincluding the peripheral environment.

FIG. 13 is an explanatory diagram conceptually illustrating aconfiguration of the finger-eye-camera measurement section 32.

In FIG. 13, the finger eye camera is constituted by a high speed camera32 a and an omnidirectional mirror 32 b, and is configured to take areflected image of the omnidirectional mirror 32 b by the high speedcamera 32 a.

The high speed camera 32 a is provided for each finger of the robot hand34 b positioned at the tip of the manipulator 31, and theomnidirectional mirror 32 b is provided so as to be opposed to thefinger and the high speed camera 32 a.

The high speed camera 32 a inputs an image taken by the robot hand 34 b,as the operation monitoring information, in the section 6 for teachingtask and the controller 30, and inputs the image via the controller 30to the error-inducing-task restraining section 5 and the operationmastering section 7.

The image information acquired from the high speed camera 32 a not onlyhelps the teaching task by a manual operation of the operator and thetemporary halt recovery operation, but is also used as an image inputfor sensor feedback control in the controller 30.

The image information can be used for autonomous and semi-autonomousconfiguration of the various tasks in this way.

The finger eye camera can be used to promote the observation of the handtip state of the manipulator 31, and to adaptively select lateral imageinformation, thereby using the image information as an image for sensorfeedback even if an intended object in the fingertip direction cannot bedirectly observed due to an occlusion.

Moreover, the finger eye camera can accurately measure a distancedistribution in a periphery of the fingertip of the manipulator 31 byproviding a near distance measurement function.

Even if the finger eye camera is used in conjunction with a cameraprovided in the environment, the same effect is provided.

A detailed description is now given of the image processing section inthe controller 30 of FIG. 1 for a path guidance by means of the sensorfeedback control.

Conventionally, feedback control according to a residual in a singleimage plane acquired from a single camera is basically carried out inthe feedback (visual feedback) using an image, and as the distanceinformation, simple distance measurement information based on a constantcondition or an object size is used.

However, highly precise feedback control in the three-dimensional spaceis difficult using those methods.

In view of the above, Example 1 of the present invention solves thisproblem by means of the feedback of an image/distance hybrid type usinga two-dimensional image feedback having a high processing rate and adistance/attitude data feedback having a slightly low processing rate.

Moreover, the semi-autonomous mode in which, while a certain attitude isautonomously maintained with respect to an intended object, aninstruction to move in a certain direction is given to other axes, or aninstruction by the operator is reflected on other axes is realized by,for example, using normal line information of the distance.

Further, according to Example 1 of the present invention, there isprovided a gripping position/attitude measurement function forrecognizing gripping position/attitude of an intended object in agripped state based on the image information acquired by the finger eyecameras.

A constraint on an attitude is generally imposed in a state in which apart is gripped, and thus a point that the image processing section inthe controller 30 can recognize the position/attitude of the object bymeans of relatively simple image processing such as three-dimensionalmatching processing limited to search specification using edgeprocessing is used.

As a result, it is possible to maintain position/attitude accuracieseven if the attitude of a part changes during the gripping work.

Input/output information for the sensor feedback function (controller30) based on the finger-eye-camera measurement section 32 of the imagemeasurement/recognition section is illustrated in an explanatory diagramof FIG. 14.

Specifically, the input information includes a camera image, neardistance data, reference model data, end point model data, andcalibration data (internal, external), and the output informationincludes target deviation data.

On the other hand, input/output information for the grippingposition/attitude measurement function (controller 30) based on thefinger-eye-camera measurement section 32 is illustrated in anexplanatory diagram of FIG. 15.

Specifically, the input information includes the camera image, thereference model data, and the calibration data (internal, external), andthe output information includes position/attitude data of an intendedobject.

A detailed description is now given of the three-dimensional recognitionsection 33 of the image measurement/recognition section of FIG. 1.

Dense distance distribution data is generally obtained by projecting apatterned light beam according to an active three-dimensionalmeasurement method, and thus a distinctive feature such as a cutoutfeature is not necessary on the intended object itself. Therefore, moregeneral position/attitude recognition is possible.

However, it is necessary to efficiently process a large amount ofthree-dimensional data, and three-dimensional position/attituderecognition algorithms presently under development are not considered tobe sufficient in terms of practical performances such as a processingtime.

Up to now, a position/attitude recognition algorithm (three-dimensionalmatching method) based on a three-dimensional distance distributionmodel illustrated in FIG. 16 has been produced on a trial basis in thecompany which the inventor belongs to.

FIG. 16 is an explanatory diagram illustrating, along with a blockdiagram, an example of three-dimensional measurement/recognitionprocessing.

In FIG. 16, an object 40 to be measured which is formed of a part of apiping, for example, is measured by a small three-dimensional sensor 41including a laser device and a three-dimensional camera.

First, the three-dimensional sensor 41 irradiates the object 40 to bemeasured with laser light while scanning the laser light, and takes animage of the object 40 to be measured by the three-dimensional cameraswhile measuring distances to the object 40 to be measured.

The three-dimensional sensor 41 then calculates a distance distributionof the object 40 to be measured, and represents the distance values intone on the image data, thereby acquiring a three-dimensionalmeasurement result 42 of the object 40 to be measured.

Finally, the three-dimensional sensor 41 carries out recognitionprocessing for the position/attitude of the object 40 to be measured,and superimposes a result of fitting a model (bright area) on themeasured data (dark area), thereby acquiring a three-dimensional model43 of the object 40 to be measured. FIG. 16 illustrates an example ofacquiring a recognition result of a plurality of objects to be measured.

However, in Example 1 of the present invention, it is desirable tofurther enhance a practical performance in terms of high speed androbustness for an application to a task of picking parts from stackedpart supply boxes (trays) and in a kitting state.

Thus, according to Example 1 of the present invention, an offlineanalysis of a search tree representing an attitude hypothesis forincreasing an efficiency, a reduction of the processing time by means ofa high-speed architecture on a computer, and the like are carried out inthe three-dimensional attitude recognition processing.

Further, if a lack of measured data or the like occurs, recognitionprocessing is carried out using only information on a measured partwhile ambiguity is remaining in terms of the position/attituderecognition, and the intended object in the gripped state is measuredagain after the picking task, thereby determining a grippingposition/attitude, resulting in systematic realization of a robustpicking function.

It is necessary for an operator to have abundant knowledge about imageprocessing if an image processing algorithm is designed in aconventional system. Moreover, though efficient processing is enabled bycombining two-dimensional processing and three-dimensional processingfor the three-dimensional recognition, more knowledge about the designis required.

According to Example 1 of the present invention, bidirectionalcooperation with the respective functional configuration sectionsrelating to the temporary halt handling is thus provided so that aproper algorithm design is carried out by combining two-dimensionalprocessing and three-dimensional processing according to a shape andtexture characteristics of the intended object, thereby supporting thedesign task carried out by the operator.

As the three-dimensional data used in Example 1 of the presentinvention, it is assumed that data obtained from three-dimensionalsensors having necessary specifications such as three-dimensional dataacquired by a compact three-dimensional range finder developed by thecompany to which the inventor belongs and the above-mentioned finger eyecamera is used.

Input/output information for the three-dimensional recognition section33 of the image measurement/recognition section is illustrated in anexplanatory diagram of FIG. 17.

Specifically, the input information includes the distance distributiondata, the camera image, the reference model data, and the calibrationdata (internal, external), and the output information includesposition/attitude data of an intended object.

Moreover, input/output information for an algorithm design supportsection: the image measurement/recognition section is illustrated in anexplanatory diagram of FIG. 18.

Specifically, the input information includes surface characteristicdata, three-dimensional shape data of an intended object, the cameraimage, and the calibration data (internal, external), and the outputinformation includes success/failure of recognition and an algorithmdesign plan.

A detailed description is now given of the universal hand of themanipulation device group 34 and the hand library 8 storing theoperation sequences of gripping tasks from the manipulation device group34.

Though effectiveness of each of the above-mentioned functionalconfiguration sections is exerted only if it is applied to the actualphysical world, the effectiveness also holds true for the error recoveryoperation in addition to normal task operations.

Specifically, even if operation strategies in the normal state and theerror state are designed, it is meaningless if physical means is notprovided for realizing the strategies. Moreover, if a form of a physicalaction element is not known, a correct action plan itself cannot beconstructed.

Therefore, end effectors typified by a hand and drive technologiestherefor (manipulation technologies) are indispensable as elementtechnologies, and have large influence in both directions on each of theabove-mentioned functional configuration sections relating to thetemporary halt handling.

A conventional end effector of a production robot often uses a gripperof a simple open/close type driven pneumatically, and dedicated handsare respectively provided according to the shape of a workpiece to begripped.

Though the operation strategy is simple in the conventional dedicatedhand system, there are problems in that design/production periods arelong, the cost is high, and the number of hands required for adapting tovarious types of products is enormous.

In contrast, according to Example 1 of the present invention, as asolution in place of the dedicated hands, a universal hand may beprovided for the cell production robot system.

However, development of a single all-purpose universal hand which canhandle all assembly tasks is not practical, and a hand which is not morecomplex than necessary should be designed according to intended tasks.

Moreover, for the universal hand of multiple degrees of freedom, it isnot sufficient only to design a mechanism thereof, but it is necessaryto consider a gripping strategy and the like up to a point of gripping aworkpiece.

Then, there is provided a framework in which the hand library 8 isgenerated by forming shapes of parts to be handled by the universalhand, fingertip positions of the universal hand suitable for grippingthe parts, gripping strategies for surely gripping workpieces in aspecified attitude even if the positions of the workpiece vary more orless, and specific hand mechanisms for realizing the fingertip positionsand operations of the universal hands into a library, information on anintended assembly task is input, and a universal hand mechanism suitablefor the assembly task is consequently presented.

FIG. 19 is an explanatory diagram graphically illustrating a concept ofthe hand library 8.

In FIG. 19, the hand library 8 associates a gripping strategy softwareprogram of the universal hand with a target task including an assemblytask and kitting, and organizes the associations as a library.

It is conceived that the gripping positions and the gripping strategiesare determined in terms of the form closure and the force closure.

The hand library 8 can be used for the action planning and the teachingtask by cooperating with each of the functional configuration sectionsrelating to the temporary halt handling in addition to for the mechanismdesign.

The information on the positions of the gripping points are made intothe library according to a given part shape and other conditions in thehand library 8, and it is thus not necessary to teach operations of thehand during an actual teaching task. Moreover, even if errors inpredetermined ranges are contained in initial position/attitude of apart, it is possible to largely reduce a period required for theteaching by organizing robust gripping strategies, which finally attainprescribed position/attitude, into a library.

As described above, the industrial robot system according to Example 1of the present invention includes the robots (robot system 3) having themanipulator 31 and the hand (manipulation device group 34), is anindustrial robot system used for a production system for assembling aproduct which is an object to be produced, and includes the actionplanning section 4 for temporary halts for generating the taskinformation and a first work path (work path including approximatecoordinates) for addressing a temporary halt which constitutes anobstacle to the teaching task when a production line is started up andadjusted, and the unattended continuous operation, and theerror-inducing-task restraining section 5 for generating the errorinformation (error occurrence probability information and causes forerror occurrence) used for restraining tasks inducing errors based onthe task information.

Moreover, the industrial robot system according to Example 1 of thepresent invention includes the section 6 for teaching task forgenerating a second work path (refined work path) based on the firstwork path and the error information, and the operation mastering section7 for generating a third work path optimized for the robot (optimizedwork path) based on the second work path.

Moreover, the industrial robot system according to Example 1 of thepresent invention includes the hand library 8 formed by associating theassembly task of the robot and the control software with each other, theoptimum-task-operation generating section 11 for generating operationsequences of specific tasks, the specific task library 9 for storing theoperation sequences of specific tasks, the error-recovery-task teachingsection 12 for teaching an error recovery task according to an errorstate based on the operation history in the section 6 for teaching task,and the error recovery library 10 for storing the error recovery tasks.

Further, the industrial robot system according to Example 1 of thepresent invention includes the finger-eye-camera measurement section 32and the three-dimensional recognition section 33 for generating theoperation monitoring information of the robot, and inputting thegenerated information to the error-inducing-task restraining section 5,the section 6 for teaching task, and the operation mastering section 7,and the controller 30 for controlling the robot based on the second andthird work paths and the operation monitoring information.

The action planning section 4 for temporary halts generates the firstwork path based on the configuration information of the productionsystem and the object to be manufactured (product design data andproduction facility data 1), each pieces of information stored in thehand library 8, the specific task library 9, and the error recoverylibrary 10, and the error information from the error-inducing-taskrestraining section 5.

The error-recovery-task teaching section 12 calculates the errorrecovery information for components including the robot in advance basedon the cause for error occurrence and the operation history from thesection 6 for teaching task.

Moreover, the action planning section 4 for temporary halts, the section6 for teaching task, and the operation mastering section 7 generate theprogram information including the third work path required for teachingthe robot from the configuration information of the production systemand the object to be manufactured.

In this way, the reduction of the installation/adjustment period for theproduction system using the industrial robots, and the extension of theerror-free continuous operation period after the operation starts can berealized by calculating, in advance, the error recovery method for eachof the components including the robot which can take multi-value statefrom the configuration information on the production system and theobject to be manufactured, and by generating the information requiredfor teaching the robot from the configuration information on theproduction system and the object to be manufactured.

Example 2

The general example in which most of the respective functionalconfiguration sections 4 to 12 are implemented as so-called software ona personal computer is described in Example 1 (FIG. 1). However, controldevices for production facilities including personal computer OSes,various real time OSes, and no OS are mixed in the real world of thefactory automation, and it is thus necessary to use software serving asglue for data exchange, data exchange, data communication, dataexchange, and media conversion in a distributed environment ofintelligent module software implemented as software on a personalcomputer such as the RT (robot technology) platform and various controldevices. Therefore, mashup sections 51 and 52 may be provided asillustrated in FIG. 20.

FIG. 20 is an explanatory diagram conceptually illustrating anindustrial robot system according to Example 2 of the present invention,and illustrating a positional relationship of the mashup section 51 forrealizing the cooperation of each of functional configuration elementsin a production system of the real factory automation.

In FIG. 20, the mashup sections 51 and 52, an upper information system53, and programmable logic controllers (PLCs) 54 and 55 are connected toa network 50, and control panels 56 and 57 are respectively connected tothe PLCs 54 and 55. Moreover, the PLCs 54 and 55 are respectivelyprovided with robot controllers (RCs) 58 and 59.

The mashup sections 51 and 52 are functional configuration sections on apersonal computer, one mashup section 51 is used for the system startup,and the other mashup section 52 is used for the system operation.

The mashup sections 51 and 52 are each constructed by a configurationsetting function (A1), a PLC/RC conversion function (A2), an executionkernel section (A3), and a safety function (A4).

The configuration setting function (A1) is a setting tool, isconstructed by a configuration provided with a GUI, and has a functionof collecting data on an arrangement of dedicated controllers for FA andconfigurations of devices in advance, and referring to the collecteddata when each of the functional configuration sections (A2) to (A4) isexecuted.

The PLC/RC conversion function (A2) has a function of distributingprograms intended for FA controllers to the FA controllers specified bythe configuration setting function (A1) after programs for controllersof the production facilities including the robots are developed usingthe functional configuration section on a personal computer, distributesthe programs converted to a sequence language to the PLCs 54 and 55, anddistributes the programs converted to a robot language to the RCs 58 and59.

Moreover, the PLC/RC conversion function (A2) also distributes programportions used for synchronizing operations among the control devicesafter assigning specific physical contact numbers or logical variablenumbers.

The execution kernel section (A3) marshals operations of the functionalconfiguration elements used for the execution on the personal computer,thereby administrating the overall execution of operations, andadministrates interfaces if each of the controllers and the functionalconfiguration elements on the personal computers cannot directlycommunicate with each other due to difference in OS or the like.

The safety function (A4) has a bridging function for exchange of aninterlock signal between the functional configuration element on thepersonal computer and a so-called safety system constructed as hardware.

As described above, the industrial robot system according to Example 2of the present invention includes the mashup sections 51 and 52 forrealizing the cooperation with the action planning section 4 fortemporary halts, the error-inducing-task restraining section 5, thesection 6 for teaching task, the operation mastering section 7, the handlibrary 8, the optimum-task-operation generating section 11, thespecific task library 9, the error-recovery-task teaching section 12,the error recovery library 10, and the controller 30. Thus, even ifindustrial robots are constructed by general-purpose personal computers,dedicated FA controllers, and a safety system, it is possible to reducethe installation/adjustment period of a production system using theindustrial robots, and to extend the no-error continuous operationperiod after the operation starts.

Example 3

Though an offline teaching section including an error detection functionfor detecting a task error, and also including the functions of theerror-inducing-task restraining section 5 and the hand library 8 may beprovided as illustrated in FIGS. 21 to 24, which is not particularlymentioned in Example 1 (FIG. 1).

The error detection function for detecting a task error and the errorrestraining function for restraining an error from occurring areeffective, and Example 3 of the present invention thus includes theoffline teaching section including these functions.

The offline teaching section includes an error-recovery-leveldetermination function and an error occurrence risk analysis function inaddition to the error detection function, the error restrainingfunction, and the hand library function.

The error detection function of the offline teaching section supportsbuilding of a task error detection logic from various types of sensorinformation (robot position, time limit, joint torque, force sensoroutput value, and image sensor output value). A serious error for whichan emergency stop is required is detected by setting a limit valuedirectly to the sensor output value.

The error-recovery-level determination function of the offline teachingsection uses a conditional branch based on the sensor information tosupport determination of an error level to be used for the errorrecovery and embedment of check points for recovery.

The error-occurrence-risk analysis function of the offline teachingsection is a function of analyzing a risk of unwanted incident, and isused in order to restrain a task error of a robot on this occasion.Specifically, if there are a plurality of candidates for work sequencesand task attitudes of a robot, risks of occurrence of various errors arecalculated as probabilities by statistically processing past errorinformation for each candidate, and the calculated values are comparedwith each other to select a task, thereby enabling frontloading therestraint of the error occurrence.

The error information used for the analysis includes sensor informationrecorded upon the error recovery, and recorded descriptions of errors.

Input/output information for the hand library function of the offlineteaching section is illustrated in an explanatory diagram of FIG. 21.

Specifically, the input information includes the part shape, and theoutput information includes gripping possibility, and grippingposition/attitude values.

Input/output information for the error detection function of the offlineteaching section is illustrated in an explanatory diagram of FIG. 22.

Specifically, the input information includes the various types of sensorinformation (robot position, time limit, joint torque, force sensoroutput value, and image sensor output value), and the output informationincludes the task error detection logic.

Input/output information for the recovery-level determination functionof the offline teaching section is illustrated in an explanatory diagramof FIG. 23.

Specifically, the input information includes the work sequence diagramillustrating work sequences, and the output information includes a worksequence diagram in which check points for determining an error leveland for recovery used upon the error recovery have already beenembedded.

Input/output information for the error-occurrence-risk analysis functionof the offline teaching section is illustrated in an explanatory diagramof FIG. 24.

Specifically, the input information includes the work sequence diagramillustrating work sequences, the work sequence diagram including errorrecovery sequences, information such as the basic design specificationsof the facilities and the robots, and the teaching data by the operator.

Moreover, the output information includes the transition information onthe error occurrence probability based on the work sequence, theestimated result/warning of error occurrence probability, and theinformation on causes/countermeasures for error occurrence.

As described above, the industrial robot system according to Example 3of the present invention includes the offline teaching section havingthe error detection function, and the offline teaching section has thefunctions of the error-inducing-task restraining section 5 and the handlibrary 8. It is thus possible to reduce the installation/adjustmentperiod of a production system using the industrial robots and to extendthe no-error continuous operation period after the operation starts.

Example 4

A visual I/F and a teleoperation function may be provided for thesection 6 for teaching task, which is not particularly mentioned inExample 1 (FIG. 1).

It is efficient to use a teleoperation section to realize intuitiveoperation on the teaching pendant, thereby reducing the operationperiod, and quickly switching a machine type during the teaching.

The teleoperation section according to Example 4 of the presentinvention uses the section 6 for teaching task as the visual I/F, andincludes a simplified sense feedback function, a refining function forteaching task using sensor information, an autonomous-control-law-designsupport function using sensor information history, and a hybridcooperative teaching function of autonomous control/manual operations.

The simplified sense feedback function realizes a simplified sensefeedback which returns a vibrator vibration or the like if a contact isdetected according to sensor information during a jog operation.

The refining function for teaching task using sensor information recordsmeasured values of the force sensor and the image sensor during theteaching task carried out by the operator, and statistically processesoperation history recorded for a plurality of times, thereby acquiringaverages and eliminating error values, resulting in generation of arefined teaching path.

The autonomous-control-law-design support function using sensorinformation history provides a result obtained by statisticallyprocessing sensor information history acquired during an operation bythe operator as a support function for designing autonomous controllaws.

The hybrid cooperative teaching function of autonomous control/manualoperations applies a designed autonomous control law only in a specificdirection of movement, and enables manual operations for otherdirections of movement, thereby realizing a teaching support function.

As described above, according to Example 3 of the present invention, thesection 6 for teaching task includes the visual I/F and theteleoperation function, can thus reduce the installation/adjustmentperiod of a production system using industrial robots, can extend theno-error continuous operation period after the operation starts, canreduce the operation period further, and can quickly switch the machinetype.

Example 5

An error recovery section cooperating with the action planning section 4for temporary halts, the section 6 for teaching task, theerror-recovery-task teaching section 12, and the error recovery library10 may be provided, and an action control function, a function ofenhancing the section for teaching task, and a teleoperation functionmay be added to the error recovery section as illustrated in FIGS. 25 to27, which is not particularly mentioned in Example 1 (FIG. 1).

The checkpoint function for detecting error occurrence is embedded in anoperation sequence of a robot output from the action planning section 4for temporary halts in the action planning section 4 for temporaryhalts, and high productivity brought about by a long-term stableoperation and a quick recovery upon a failure can thus be realized byenabling recovery action control from the detected error state by usingthe check point function.

Specifically, according to Example 5 of the present invention, an errorstate is recognized from various types of sensor information (robotposition, time limit, joint torques, force sensor, and image sensor),and action control is carried out according to a degree of errorseriousness. For an error for which an emergency stop is required,immediate stop is carried out by directly setting a limit value to thesensor output value.

An industrial robot system according to Example 5 of the presentinvention includes the error recovery section having the action controlfunction (B1), the function of enhancing the section for teaching task(B2), and the teleoperation function (B3) for surely carrying out theabove-mentioned operation.

The action control function (B1) of the error recovery section switchesamong the following three levels (M1) to (M3) of an error recoveryoperation mode according the degree of error seriousness, and assists aprocedure for restart of production by synchronizing respective robotsand respective production facilities with one another from an arbitrarystop position after recovery from an error state in each of the modes(M1) to (M3).

In the winding-back automatic recovery mode (M1), an automatic recoveryis carried out from a present position at which an error is detected toan immediate previous checkpoint in a winding back manner.

In the operator recovery operation (teleoperation) mode (M2), an errorstop signal is generated, thereby calling the operator, and the operatorprovides the error recovery operation by the teleoperation. If arecovery to a check point which is not adjacent to the present positionis required, the system supports the recovery while synchronizingrespective robots and production facilities with one another.

In the operator recovery operation (manual operation) mode (M3), anerror stop signal is generated, thereby calling the operator, and theoperator takes a step such as directly removing a workpiece which hascaused the error from a task location. If a recovery to a check pointwhich is not adjacent to the present position is required, the systemsupports the recovery while synchronizing respective robots andproduction facilities with one another.

The function of enhancing the section for teaching task (B2) of theerror recovery section carries out an intuitive display of complexinformation obtained based on the force sensor and the camera image, anddisplays the force sensor information visualized on the finger eyecamera image, thereby supporting an intuitive operation. Moreover, thefunction of enhancing the section for teaching task (B2) increasesvisibility of the finger eye camera image displayed on a display, andcarries out digital zooming, edge-enhanced image generation, andcontrast improvement.

The teleoperation function (B3) of the error recovery section uses thesection 6 for teaching task as the visual I/F, thereby carrying out thesimplified sense feedback. In other words, when a contact is detectedaccording to sensor information during jog operation, the simplifiedsense feedback is realized by, for example, returning a vibratorvibration or the like.

Input/output information for the action control function of the errorrecovery section is illustrated in an explanatory diagram of FIG. 25.

Specifically, the input information includes the work sequence diagramand the degree of error seriousness, and the output information includesthe error recovery operation mode.

Input/output information for the function of enhancing the section forteaching task of the error recovery section is illustrated in anexplanatory diagram of FIG. 26.

Specifically, the input information includes the various types of sensorinformation (robot position, time limit, joint torque, force sensoroutput value, and image sensor output value), the work path, and anoperated quantity by an operator, and the output information includes animage displaying combined information.

Input/output information for the teleoperation function of the errorrecovery section is illustrated in an explanatory diagram of FIG. 27.

Specifically, the input information is the operated quantity of theoperator, and the output information is asimplified-sense-feedback-control operated quantity.

As described above, the industrial robot system according to Example 5of the present invention includes the error recovery section cooperatingwith the action planning section 4 for temporary halts, the section 6for teaching task, the error-recovery-task teaching section 12, and theerror recovery library 10. The error recovery section includes theaction control function, the function of enhancing the section forteaching task, and the teleoperation function, and thus it is possibleto reduce the installation/adjustment period of a production systemusing the industrial robots and to extend the no-error continuousoperation period after the operation starts. It is also possible torealize a stable operation for a long period and high productivity byquick recovery upon a failure.

Example 6

A recognition section cooperating with the section 6 for teaching task,the operation mastering section 7, the finger-eye-camera measurementsection 32, and the three-dimensional recognition section 33 may beprovided, and an object recognition function for part picking, a hybridvision correction function, a vision function for error detection, and arecognition application building support function may be added to therecognition section as illustrated in FIGS. 28 to 31, which is notparticularly mentioned in Example 1 described above.

Quick machine-type switching and high productivity are realized byproviding the recognition section required for the teaching, theexecution operation, and the error detection in Example 6 of the presentinvention.

It is assumed that the finger eye cameras and fixed camera on theenvironment side (including stereoscopic camera configuration) areprovided for imaging, and a three-dimensional sensor such as athree-dimensional range finder is used according to necessity.

The recognition section according to Example 6 of the present inventionincludes an object recognition function for part picking (C1), a hybridvision correction function (C2), a vision function for error detection(C3), and a recognition application building support function (C4).

The object recognition function for part picking (C1) of the recognitionsection constitutes a recognition function for picking an object from apart box by using the three-dimensional recognition section 33, andincludes a function of checking interference upon the picking and aposition correction function of a gripped object in the part grippingstate.

The hybrid vision correction function (C2) of the recognition sectionrealizes a two-dimensional/three-dimensional hybrid vision correctionfunction by using the three-dimensional recognition section 33.Moreover, the hybrid vision correction function (C2) realizes a functionof moving to a task point based on a three-dimensional position/attitudeoutput acquired by using the three-dimensional recognition section 33after receiving a relative positioning instruction in an image, and alsorealizes a semi-autonomous moving function along with a manual movementinstruction by moving while maintaining constant relativeattitude/relative distance with respect to a constraining surface.

The vision function for error detection (C3) of the recognition sectionuses an image recognition module which is generally available, such as acommercially available one, and uses an image acquired by the finger eyecameras, thereby providing a check function for a part shape. On thisoccasion, error detection is carried out based on a result of applyingstatistical processing based on a plurality of pieces of part shape datain the checking.

The recognition application building support function (C4) of therecognition section provides a function for supporting a user in easilybuilding applications, such as model registration, parameter setting,calibration, and the like, when execution functions for a picking task,a visual correction task, and error detection using the respectivefunctional configuration sections (C1) to (C3) are built.

Input/output information for the object recognition function for partpicking (C1) (recognition module) of the recognition section isillustrated in an explanatory diagram of FIG. 28.

Specifically, the input information includes sensor output data,parameters, and model information, and the output information includes arecognition result for object picking from a part box, an interferencecheck result, and a position correction value of a gripped object.

Input/output information for the hybrid vision correction function (C2)(correction module) of the recognition section is illustrated in anexplanatory diagram of FIG. 29.

Specifically, the input information includes the sensor output data, arelative positioning instruction in a two-dimensional image, andthree-dimensional position/attitude values, and the output informationincludes a movement trajectory to a task point, and a movement quantity.

Input/output information for the vision function for error detection(C3) (vision module) of the recognition section is illustrated in anexplanatory diagram of FIG. 30.

Specifically, the input information includes an image acquired from thefinger eye cameras and a plurality of pieces of part shape data, and theoutput information includes an error detection output.

Input/output information for the recognition application buildingsupport function (C4) (building support module) of the recognitionsection is illustrated in an explanatory diagram of FIG. 31.

Specifically, the input information includes the model information, theparameter information, and the calibration data.

Moreover, the output information includes recognition processingparameters, model data of an object to be recognized, and datarepresenting a recognition processing sequence.

As described above, the industrial robot system according to Example 6of the present invention includes the recognition section cooperatingwith the section 6 for teaching task, the operation mastering section 7,the finger-eye-camera measurement section 32, and the three-dimensionalrecognition section 33. The recognition section includes the objectrecognition function for part picking, the hybrid vision correctionfunction, the vision function for error detection, and the recognitionapplication building support function, and thus it is possible to reducethe installation/adjustment period of a production system using theindustrial robots, and to extend the no-error continuous operationperiod after the operation starts. It is also possible to realize quickswitching of the machine type and high productivity.

It is to be understood that the configurations of Examples 1 to 6 can bearbitrarily combined for application, and that this provides overlappingactions and effects.

Example 7

Further, in Example 1 described above (FIG. 1), the operation masteringsection 7 and the various types of libraries 8 to 10 are provided, andalso, the finger-eye-camera measurement section 32 and thethree-dimensional recognition section 33 are provided in the robotsystem 3. However, those components may be omitted, and a configurationillustrated in FIG. 32 may be provided.

FIG. 32 is a block configuration diagram illustrating an industrialrobot system according to Example 7 of the present invention. The likecomponents are denoted by like numerals or by like numerals followed by“A” as for those described before (refer to FIG. 1).

Referring to FIG. 32, the industrial robot system includes the productdesign data and production facility data 1 (including part connectioninformation, geometric shape data, and facility layout data) producedand prepared in advance by means of a three-dimensional CAD, and a robotsystem 3A installed on a production line.

Moreover, the industrial robot system includes an action planningsection 4A for temporary halts, an error-inducing-task restrainingsection 5A, a section 6A for teaching task, a controller 30A, and amanipulator 31A as a configuration relating to the product design dataand production facility data 1 and the robot system 3A. The controller30A and the manipulator 31A are provided in the robot system 3A.

The action planning section 4A for temporary halts generates a worksequence diagram including error recovery sequences and a work pathsincluding approximate coordinates based on the part connectioninformation, the geometric shape data, and the facility layout data fromthe product design data and production facility data 1, and inputs thework sequence diagram and the work paths to the section 6A for teachingtask.

Moreover, the action planning section 4A for temporary halts is mutuallyrelated to the error-inducing-task restraining section 5A, and inputscandidates of the task order to the error-inducing-task restrainingsection 5A and also receives the error occurrence probabilityinformation from the error-inducing-task restraining section 5A.

The section 6A for teaching task generates refined work paths (robotprogram before mastering) based on the work sequence diagram (includingerror recovery sequences) and the work paths (including approximatecoordinates) from the action planning section 4A for temporary halts,and the operation monitoring information from the robot system 3A, andinputs the refined work paths to the controller 30A.

The controller 30A provides control for driving the manipulator 31Abased on the robot programs and the operation monitoring informationfrom the manipulator 31A.

A description is now given of an operation according to Example 7 of thepresent invention illustrated in FIG. 32.

As general design steps for a product by a designer, the structuraldesign of a product to be manufactured, and the layout design for cellsused to manufacture the product are first carried out as describedbefore.

As a result, the part connection information representing a connectionorder of parts constituting the product (part configuration treediagram), the product design data such as the geometric shape data ofthe parts, the facility layout data within the cells, and the productionfacility data such as specifications of the robots are obtained.

The system operation according to Example 7 of the present inventionstarts from a state in which those results of the design task by adesigner are available.

A description is now given of the system operation according to Example7 of the present invention when the production facility starts.

The product design data and production facility data 1 are input to theaction planning section 4A for temporary halts in a first phase.

As a result, a product production task is decomposed into a sequence ofsmaller tasks, and each of these tasks are assigned to each of thefacilities in the cell, and a task order is generated based on the partconnection information in the action planning section 4A for temporaryhalts. On this occasion, when the task is decomposed and the task orderis determined, if a candidate of the task order is given to theerror-inducing-task restraining section 5A, the error occurrenceprobability information for the given task is returned, and a task orderhaving a low risk of temporary halt is thus selected. Note that, theerror occurrence probability is updated at any time by the operationmonitoring information while production is being carried out.

Moreover, the action planning section 4A for temporary halts determinesthe respective tasks and the task order, and generates a “work sequencediagram including error recovery sequences” including check points forexamining an occurrence of a temporary halt during the task, a recoverypoint at which the task can be resumed when a check point is not passed,a recovery path for returning to the recovery points, a via point foravoiding an obstacle, work paths connecting between the respectivepoints to each other, and a sequence describing the execution order ofthe respective paths, and a synchronization point at which other robotsand devices are caused to wait.

Moreover, an attribute label is attached to each of the work paths.

Specifically, the labels include “movement between two points”,“movement to via pint”, “task accompanying action by end effectordevice”, “task according to sensor feedback control such as approachingmovement immediately before/after gripping part or task operationoptimized in advance”, and “during recovery sequence from temporaryhalt”. Note that, a plurality of labels may be attached to one workpath.

On this stage, the work sequence diagram includes only the labeled workpaths and the respective points, and does not include contents of eachof the work paths.

Note that, the content of the work path includes position and attitudecoordinates at both ends of the path and via points thereof (a pluralityof via points are added according to necessity) and a specification ofmethod of movement between the coordinates (such as control law andinterpolation method).

Moreover, the task operation optimized in advance includes tips forpreventing task errors, and quickly and flexibly carrying out the task.

As a next phase, rough contents of the work paths for each of the tasksare generated by using facility layout data within the cells and taskorder data in the action planning section 4A for temporary halts.

For example, if there are a part placement location and a work bench ina production cell, and a work path of a specific trajectory fortransporting a part is considered for a task for transporting a partfrom the part placement location to the work bench, a robot mayinterfere with surrounding objects, and it is eventually necessary toset precise values to instances of each of the work paths.

Each of the work paths is generated at a precision of approximately 5 cmon this occasion, and a reference attitude for gripping an object(relative attitude between the part and the hand) is determined in thesection 6A for teaching task on a later stage by an operator usingteaching task input means.

The above-mentioned operation is repeated for all the work paths, anddata including “work sequence diagram including error recoverysequences+work paths including approximate coordinates” is obtained asthe output information from the action planning section 4A for temporaryhalts up to this phase.

The section 6A for teaching task then starts an operation.

Refined work paths to which absolute coordinates are specified aredetermined by a teaching operator using teaching task input means bycarrying out only final positioning of important operation points suchas gripping points for the work paths including approximate coordinatesin the section 6A for teaching task.

A user interface based on the ecological interface theory displayed on apersonal computer for the teaching task or a teaching pendant presentsimportant operation points and task states on this occasion, and theoperator carries out the refining task for the position/attitude, andadds work paths according to necessity on the teaching task input meanswhile observing the presented states.

As described above, the action planning section 4A for temporary halts,the error-inducing-task restraining section 5A, and the section 6A forteaching task in cooperation with the respective pieces of design data 1generate a robot program for the robot system 3A.

That is, also in the configuration of Example 7 (FIG. 32) of the presentinvention, a robot program executable on the controller 30A includingrecovery sequences for the case of occurrence of a temporary halt can beobtained from the product design data and production facility data 1while the load on the teaching operator is reduced significantlycompared with a conventional case.

REFERENCE SIGNS LIST

1: product design data and production facility data, 2: specific taskspecification, 3, 3A: robot system, 4, 4A: action planning section fortemporary halts, 5, 5A: error-inducing-task restraining section, 6, 6A:section for teaching task, 7: operation mastering section, 8: handlibrary, 9: specific task library, 10: error recovery library, 11:optimum-task-operation generating section, 12: error-recovery-taskteaching section, 30, 30A: controller, 31, 31A: manipulator, 32:finger-eye-camera measurement section, 33 three-dimensional recognitionsection, 34: manipulation device group, 34 a: universal hand, 34 b:robot hand, 32 a: high speed camera, 32 b: omnidirectional mirror, 40:object to be measured, 41: three-dimensional sensor, 50: network, 51:mashup section, W: part to be assembled

The invention claimed is:
 1. An industrial robot system including arobot having a manipulator and a hand with a plurality of fingers, andused for a production system for assembling a product which is an objectto be manufactured, comprising: action planning circuitry for temporaryhalts to generate task information and a first work path in order toaddress a temporary halt that constitutes an obstacle to a teaching taskwhen a production line is started up and adjusted, the first work pathdefining approximated and unrefined robot movements from a start pointto an end point; error-inducing-task restraining circuitry to generateerror information used for restraining a task inducing an error based onthe task information; task-teaching circuitry to generate a second workpath specifying absolute coordinates based on the first work path, theerror information, and an input from an operator, the second work pathdefining specific and refined robot movements from the start point tothe end point; operation mastering circuitry to generate a third workpath optimized for the robot based on the second work path, the thirdwork path defining optimized robot movements from the start point to theend point; a memory that stores a hand library formed by associating anassembly task of the robot and control software with each other;optimum-task-operation circuitry to generate an operation sequence ofspecific tasks; the memory stores a specific task library for storingthe operation sequence of specific tasks; error-recovery-task teachingcircuitry to teach an error recovery task according to an error statebased on an operation history in the task-teaching circuitry; the memorystores an error recovery library for storing the error recovery task;finger-eye-camera measurement circuitry and three-dimensionalrecognition circuitry to generate operation monitoring information onthe robot, and to input the operation monitoring information to theerror-inducing-task restraining circuitry, the task-teaching circuitry,and the operation mastering circuitry, the finger-eye-camera measurementcircuitry including a different finger camera provided close to thefingertip of each finger of the hand and an omnidirectional mirror thatis opposite to each finger and finger camera so at least one of thefinger cameras can capture a reflected image of the fingertip and aperiphery of the fingertip as a partial basis for the operationmonitoring information on the robot; and a controller to control therobot based on the second work path and the third work path, and on theoperation monitoring information, wherein: the action planning circuitrygenerates the first work path based on configuration information on theproduction system and the object to be manufactured including at leastpart connection information, geometric shape data, and facility layoutdata, information stored in the hand library, the specific task library,the error recovery library, and the error information from theerror-inducing-task restraining circuitry; the task information includesat least a check point for examining a temporary halt during the task, arecovery point returning to which enables the task to resume when thecheck point is not passed, a recovery path for returning to the recoverypoint, work paths connecting the respective points to each other, and asequence describing the execution sequence of each of the work paths,and a synchronization point; the task-teaching circuitry, based on theinput by the operator, warns the operator about a task operation that islikely to cause a mistake when the second work path is being generated;the error-recovery-task teaching circuitry calculates error recoveryinformation on components including the robot based on a cause foroccurrence of the error and the operation history from the task-teachingcircuitry; and the action planning circuitry for temporary halts, thetask-teaching circuitry, and the operation mastering circuitry generateprogram information including the third work path required for teachingthe robot from the configuration information on the production systemand the object to be manufactured.
 2. An industrial robot systemaccording to claim 1, further comprising: mashup circuitry to providecooperation among the action planning circuitry for temporary halts, theerror-inducing-task restraining circuitry, the task-teaching circuitry,the operation mastering circuitry, the hand library, theoptimum-task-operation generating circuitry, the specific task library,the error-recovery-task teaching circuitry, the error recovery library,and the controller.
 3. An industrial robot system according to claim 1,further comprising: offline teaching circuitry having an error detectionfunction, wherein the offline teaching circuitry has functions of theerror-inducing-task restraining circuitry and the hand library.
 4. Anindustrial robot system according to claim 1, wherein the task-teachingcircuitry has a visual interface and a teleoperation function.
 5. Anindustrial robot system according to claim 1, further comprising: errorrecovery circuitry that cooperates with the action planning circuitryfor temporary halts, the task-teaching circuitry, theerror-recovery-task teaching circuitry, and the error recovery library,wherein the error recovery circuitry has an action control function, afunction of enhancing the task-teaching circuitry, and a teleoperationfunction.
 6. An industrial robot system according to claim 1, furthercomprising: recognition circuitry that cooperates with the task-teachingcircuitry, the operation mastering circuitry, the finger-eye-camerameasurement circuitry, and the three-dimensional recognition circuitry,wherein the recognition circuitry has an object recognition function forpart picking, a hybrid vision correction function, a vision function forerror detection, and a recognition application building supportfunction.
 7. An industrial robot system according to claim 1, furthercomprising: a fingertip force sensor provided close to the fingertip ofeach finger of the hand that measures a force and torque applied to thefingertip; and the task-teaching circuitry further warns the operatorabout the task operations that is likely to cause the mistake when thesecond work path is being generated by displaying the reflected imagecaptured by at least one of the finger cameras and displayinginformation obtained by matching data acquired by the fingertip forcesensor against a model prepared in advance and obtained by abstracting aphenomenon occurring on the robot fingertip.
 8. An industrial robotsystem including a robot having a manipulator and a hand with aplurality of fingers, and used for a production system for assembling aproduct which is an object to be manufactured, comprising: actionplanning circuitry for temporary halts to generate task information anda first work path in order to address a temporary halt that constitutesan obstacle to a teaching task when a production line is started up andadjusted, the first work path defining approximated and unrefined robotmovements from a start point to an end point; error-inducing-taskrestraining circuitry to generate error information used to restrain atask inducing an error based on the task information; task-teachingcircuitry to generate a second work path specifying absolute coordinatesbased on the first work path, the error information, and an input froman operator, the second work path defining specific and refined robotmovements from the start point to the end point; finger-eye-camerameasurement circuitry and three-dimensional recognition circuitry togenerate operation monitoring information on the robot, and to input theoperation monitoring information to the error-inducing-task restrainingcircuitry and the task-teaching circuitry, the finger-eye-camerameasurement circuitry including a different finger camera provided closeto the fingertip of each finger of the hand and an omnidirectionalmirror that is opposite to each finger and finger camera so at least oneof the finger cameras can capture a reflected image of the fingertip anda periphery of the fingertip as a partial basis for the operationmonitoring information on the robot; and a controller for controllingthe robot based on the second work path and the operation monitoringinformation, wherein: the action planning circuitry generates the firstwork path based on configuration information on the production systemand the object to be manufactured including at least part connectioninformation, geometric shape data, and facility layout data, and theerror information from the error-inducing-task restraining section; thetask information includes at least a check point for examining atemporary halt during the task, a recovery point returning to whichenables the task to resume when the check point is not passed, arecovery path for returning to the recovery point, work paths connectingthe respective points to each other, and a sequence describing theexecution sequence of each of the work paths, and a synchronizationpoint; the task-teaching circuitry, based on the input by the operator,warns the operator about a task operation that is likely to cause amistake when the second work path is being generated; and the actionplanning circuitry and the task-teaching circuitry generate programinformation including the second work path required for teaching therobot from the configuration information on the production system andthe object to be manufactured.
 9. An industrial robot system accordingto claim 8, further comprising: a fingertip force sensor provided closeto the fingertip of each finger of the hand that measures a force andtorque applied to the fingertip; and the task-teaching circuitry furtherwarns the operator about the task operations that is likely to cause themistake when the second work path is being generated by displaying thereflected image captured by at least one of the finger cameras anddisplaying information obtained by matching data acquired by thefingertip force sensor against a model prepared in advance and obtainedby abstracting a phenomenon occurring on the robot fingertip.